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ABSTRACT 

The Scholarship of Teaching and Learning (SoTL) has been described as the fastest growing 
academic development movement in higher education. As this field of inquiry matures, there 
is a need to understand how SoTL research is conducted. The purpose of our study was to 
inform this debate by investigating research approaches used in SoTL publications. We 
analysed 223 empirical research studies published from 2012 to 2014 in three explicitly- 
focused SoTL journals. We classified the studies as either qualitative, quantitative, or mixed 
methods using an analytical framework devised from existing literature on research methods. 
We found that the use of the three research designs was fairly evenly distributed across the 
papers examined: qualitative (37.2%), quantitative (29.6%), and mixed methods (33.2%). 
However, there was an over-reliance on data collection from a single source in 83.9% of papers 
analysed, and this source was primarily students. There was some, but limited, evidence of the 
use of triangulation through the use of multiple data collection instruments (e.g. survey, 
assessment tasks, grade databases). Similarly, only one-third of publications classified as mixed 
methods integrated the analysis and interpretation of the qualitative and quantitative data 
equally within the study. We conclude that current SoTL research is characterised by 
methodological pluralism but could be advanced through inclusion of more diverse 
approaches, such as close reading, and adoption of strategies known to enhance the quality of 
research, for example, triangulation and visual representation. 
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INTRODUCTION 

The Scholarship of Teaching and Learning (SoTL) has been described as the fastest growing 
academic (termed ‘faculty’ in North America) development movement in higher education (Gibbs, 
2013). Arising from a broader perspective on academic work, Boyer (1990) described four overlapping 
domains of scholarship, one being the scholarship of teaching. There has been a plethora of literature 
attempting to define, categorise, or simply make sense of SoTL. Potter and Kustra (2011) define SoTL 
as “the systematic study of teaching and learning, using established or validated criteria of scholarship, to 
understand how teaching (beliefs, behaviours, attitudes, and values) can maximize learning, and/or 
develop a more accurate understanding of learning, resulting in products that are publicly shared for 
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critique and use by an appropriate community.” Although contested and complicated, the aim of SoTL 
is to enhance instructors’ teaching practices and students’ learning experiences. 

Given the roots of SoTL in academic work, SoTL has a strong link in discipline identity (Healey, 
2000). Thus, it is unsurprising that no single specific research approach can characterize all or even the 
majority of SoTL publications (Wilson-Doenges & Gurung, 2013). An academic’s own disciplinary 
practice is seen to influence their SoTL practice (Huber & Morreale, 2002; Bass & Linkon, 2008; Chick, 
Hassel, & Haynie, 2009). The benefit of many varying disciplinary perspectives in SoTL is the potential 
for ‘pluralism’ in both methods and theories used to explore teaching in higher education (Hutchings & 
Huber, 2008). This resonates with the image of SoTL as a ‘big tent,’ inclusive of diverse approaches and 
perspectives (Huber & Hutchings, 2005). Debating the validity of one form of research over another 
across disciplines is not new for the academy and is evident within disciplines as they form and establish 
themselves. As SoTL grows and matures, the debate of‘what is research in SoTL?’ seems appropriate 
(Poole, 2012). In outlining ‘good practice principles’ of SoTL, Felten (2013) acknowledges the 
pluralism of SoTL that can reflect a variety of disciplinary ethos. Regardless of approach, Felten (2013) 
argues that “good practice in SoTL requires the intentional and rigorous application of research tools 
that connect the question at the heart of a particular inquiry to student learning” (p. 123). Methods 
should be sound and appropriately aligned to the question driving the SoTL inquiry. 

Approaches to SoTL research have been debated for over a decade (Felten, 2013). What 
research approaches and methods of data collection and analysis are used in published SoTL work? 
Interestingly, with a few exceptions, this question has not been explored in much depth. A North 
American research-intensive university reviewed ten years of their institutional SoTL activity and found 
the predominant SoTL study design relied on qualitative approaches (Hubball, Clarke & Poole, 2010). 
A study analysing three years of published work in three journals explicitly focused on SoTL reported 
that 38% of papers employed qualitative approaches, 34% used quantitative approaches, and 28% used 
mixed methods (Matthews et ah, 2013). However, the focus of the latter study was on students' 
experiences of their degree-level programs and therefore did not include a detailed analysis of 
methodological approaches. Prior to the 2014 International Society for the Scholarship of Teaching and 
Learning Conference (ISSOTL), a small group gathered to examine abstracts for talks and posters 
included in the conference program. They explored discipline of authors, data generated from students 
(e.g. perception, performance, both, neither), and level of student data (e.g. undergraduate, post¬ 
graduate) to find approximately 44% of abstracts drew on learner data (Powell, Harmon, & MacMillan, 
2015). 

As a subset of the researchers who conducted the Matthews, Divan, John-Thomas, Lopes, 
Ludwig, Martini, Motley and Tomljenovic-Berube (2013) study, we acknowledged at the time the need 
to further explore approaches to research in SoTL. SoTL is growing and maturing as a field of inquiry, 
and debates continue about what SoTL is and what constitutes SoTL research. An in-depth 
understanding of the types of research approaches SoTL scholars currently use, as featured in the 
published literature, could provide an evidential baseline to inform the debate and shape future 
discussions on what research in SoTL could or should look like. This study contributes such an 
evidential baseline, which is important for the emerging SoTL field that draws practitioners from a range 
of disciplinary boundaries. As SoTL continues to grow and mature, understanding how SoTL research is 
conducted becomes vital to engaging new SoTL practitioners. 

Conceptualising SoTL research: qualitative approach 

Qualitative research investigates the ways that individuals and groups encounter and understand 
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various experiences within natural settings (Cochrane & Oliver, 2006) and can help describe the nature 
and essence of what’s being studied (Chick, 2013). Open-ended questions are the hallmark of 
qualitative SoTL research and are explored mainly through textual (but also verbal and visual) data, 
gathered through reflective writing, focus groups, interviews, field notes, participant observations, case 
studies, or document reviews (Hart, Smith, Swars, & Smith, 2009; Hubball & Clarke, 2010; Chick, 
2013). Qualitative research allows for in-depth analysis of complex systems and experiences which 
cannot be fully captured with measurement scales and multivariate models (Plano Clark et ah, 2008) 
and is best suited to address a research problem in which the desired variables are yet unknown 
(Creswell, 2012) or when a quantifiable attribute at the data collection stage is not assigned. According 
to Castro, Kellison, Boyd, and Kopak (2010), the strengths of this methodology include a) the capacity 
for generating richly detailed accounts of human experiences (emotions, beliefs, and behaviours), and b) 
narrative accounts that can be examined within the original context in which observations occur. 

Critiques of qualitative research often include claims that it’s too subjective, it can be difficult to 
replicate, and there is a lack of transparency regarding methods and how conclusions are drawn 
(Bryman, 2012; Dreher, 1994). However, these critiques may not be of qualitative research, per se, but 
possibly of research design, implementation, and description. 

Operational definition of the qualitative approach 

For the purpose of our study, we defined the qualitative approach as one that explores a central 
phenomenon without assigning a quantifiable attribute, permitting a broad view of the participant’s 
experiences. Data may be quantified for data analysis purposes and presentation, post-data collection. 
The key principle guiding our analysis was that data were collected in words, not as numbers. 

Conceptualising SoTL research: quantitative approach 

Quantitative SoTL research can be characterized as that which seeks to confirm hypotheses, 
quantify variation, or predict causal relationships through the use of methods that directly gather 
numerical data and enable statistical analysis or some other form of quantification (e.g. percentages or 
descriptive statistics) (Meyers, 2008; Chick, 2013). Quantitative explorations primarily make use of 
closed-ended questions employing surveys, questionnaires, or structured observations (Hubball & 
Clarke, 2010; Chick, 2013). The strengths of quantitative research include accurate measurement, the 
ability to make correlations and associations, and the ability to test hypotheses. Furthermore, 
quantitative research allows for group comparisons and can take advantage of larger sample sizes to 
target a broader pool of participants (Castro et al., 2010; Creswell, 2012, p. 14). 

Like data gathered using qualitative approaches, quantitative research has weaknesses. 
Quantitative research is limited by the fact that measurements often separate information from its 
context (Castro et al., 2010) and its use often fails to make the distinction between people and social 
institutions and the “world of nature” (Bryman, 2012, p. 178). Furthermore, the use of measurements 
can provide a false sense of precision and accuracy, whilst the analysis of relationships between variables 
can create a static view of social life independent of the actual experiences of individuals (Bryman, 

2012 ). 


Operational definition of the quantitative approach 

For the purpose of our study, we defined the quantitative approach as one that investigates 
variables that are quantifiable (counted) using methods that permit straight-forward data gathering that 
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is unambiguous in interpretation. The key principle for our data analysis was that data were collected as 
numbers. 

Conceptualising SoTL research: mixed-method approach 

A third approach to research involves a blend of qualitative and quantitative methods, and is 
referred to as “mixed methods” research. Mixed research methodologies benefit from the inherent 
strengths of qualitative research (the descriptive analysis of textual, verbal, and visual artifacts) and 
quantitative research (the precision in measurement and hypothesis testing) (Castro et al., 2010; Hart, 
Smith, Swars & Smith, 2009; Hubball et ah, 2010). Quantitative and qualitative methods maybe used 
sequentially or concurrently with the goal of bringing together differing, complementary forms of data. 
Thus mixed methods research can provide a better understanding of research problems than neither 
qualitative nor quantitative approaches alone allow (Chick, Karis, & Kernahan, 2009; McKinney, 2007; 
Morse, 1991) and can provide more in-depth focus on selected interests within a study (Creswell, 

2012). Furthermore, when working with people, like students or academics, scholars have found 
advantages in combining aspects of qualitative and quantitative research, as drawing data from various 
perspectives can provide a richer overall picture of why and how a particular teaching strategy may 
influence learning (Chick, 2013). 

The extent to which researchers integrate or mix the findings and inferences (partially or fully) 
generated by the qualitative and quantitative phases of the study and the prominence afforded to each 
approach (equal status or dominant status) are considered fundamental to the design of mixed method 
research (Greene, 2007; Leech & Onwuegbuzie, 2009). Thus, mixed methods research can require 
extensive time and effort in order to effectively implement both approaches and may necessitate diverse 
methodological training, or collaboration within a team of researchers to ensure appropriate use of each 
approach (Creswell, 2012). 

Operational definition of mixed methods approach 

For the purpose of our study, we defined the mixed method approach as one where aspects of 
both qualitative and quantitative research are used at any stage of the study based on the operational 
definition used by Hart, Smith, Swars, and Smith (2009). A key guiding principle for our analysis was 
that data were collected as numbers and as text. Where the data were collected as text only, this is 
qualitative, even if the authors quantify post-collection. The extent to which SoTL researchers brought 
together qualitative and quantitative components of their study and their relative importance was 
analysed subsequent to classification of a study as mixed methods. 

Research question 

To address our research question of what types of research approaches are commonly used by 
SoTL researchers, we considered specifically the following: 

1. How prevalent is quantitative research only, qualitative research only, or mixed methods 
research amongst empirical SoTL articles published in a collection of prominent and 
explicitly-focused SoTL journals from 2012 to 2014? 

2. What data do SoTL researchers commonly gather, and when and how do they collect 
and analyze these data? 

3. When mixed methods research is used, how are qualitative and quantitative approaches 
integrated and/or balanced within these articles? 
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METHOD 

The focus of this study was to explore the prevalent methodological approaches used by scholars 
pursuing research in SoTL. Our approach was to conduct a literature review that was grounded in 
awareness of the challenges and limitations associated with “defining the literature” and then reviewing 
that defined body of literature drawing on conceptually rich notions of methodological research 
approaches (Kennedy, 2007). 

Defining the body of literature 

Our body of literature for the purposes of this study included the assessment of 267 empirical 
research studies published from 2012 to 2014 in three SoTL journals: the InternationalJournal for the 
Scholarship of Teaching and Learning (IJSOTL), the InternationalJournal of Teaching and Learning in 
Higher Education (IJTLHE), and the Journal of the Scholarship of Teaching and Learning (JO SOTL). 
The journals were selected specifically for their clear affiliation with the SoTL movement. Articles for 
inclusion were those with an empirical focus that could clearly be identified as research-based articles. 
On this basis, we excluded 126 articles that lacked an empirical focus including those written as opinion 
pieces, book reviews, and editorials. We also excluded empirical studies that were testing surveys or 
other research tools as our focus was on publications specifically collecting and reporting on data from 
SoTL studies. As such, 44 articles were excluded. Table 1 displays the number of articles in each journal, 
total number of empirical studies, and number of articles included for the study within the defined time 
frame. 

Table 1. Journal titles, total number of articles, number of empirical studies, and number of articles included in current study. 


Journal Titles 

Total Number of Articles 

Number of Empirical 
Studies 

Number of Articles 
Included for Study 

InternationalJournal of the Scholarship 
of Teaching and Learning 

162 

93 

79 

InternationalJournal of Teaching and 
Learning in Higher Education 

110 

66 

50 

Journal of the Scholarship of Teaching 
and Learning 

121 

108 

94 


393 

267 

223 


Review team 

A total of six researchers from six different higher education institutions participated in the 
analysis. Researchers came from a range of disciplines (e.g. biosciences, chemistry, communications, 
education, English) with one male researcher. The team worked at a distance with an established mode 
of online communication and experience in conducting literature reviews (Matthews et ah, 2013). 

Conducting the analysis 

Four phases of testing the analysis framework were conducted that involved each researcher 
independently analysing the same article and entering results into an online form. Differences in 
interpretation were identified and discussed, and the analysis form was revised accordingly. This 
iterative process was essential for revealing implicit differences in how the researchers conceptualised 
and classified SoTL research as being qualitative, quantitative, or mixed methods. 
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To conduct the final analysis, researchers first worked individually to read and analyse assigned 
articles (approximately 75 per researcher). Each article was read and analysed by two researchers 
independently. Then, researchers partnered and consulted on analysis for articles. Following discussion, 
the two researchers came to agreement on the classification of the article. Where researchers could not 
agree, a third researcher was consulted. Once consensus was reached about an article, the data was 
entered into an online system that captured the analysis in a standardised format. 

Analysis framework 

The purpose of our analysis framework was to systematically capture data from each article 
guided by our pre-determined research questions whilst also reducing researchers’ variability in the 
interpretation of SoTL research. Two papers were heavily drawn on to develop the analysis framework 
(Amundsen & Wilson, 2012; Hart et al., 2009). The Hart, Smith, Swars, and Smith literature review, in 
particular, guided our defining of qualitative, quantitative, and mixed method approaches. Our 
operational definitions to classify articles as utilising qualitative, quantitative, or mixed-methods are 
presented above. Our analysis framework collected demographic information about the article (e.g. 
journal, year, country of first author, volume, issue) and included the following prompts: 

1. Article was focused in single university, multiple universities in the same country, multiple 
universities in different countries, other (specify). 

2. Sources of data were students, academics, document analysis, other (specify). 

3. Data was collected from surveys, interviews, focus groups, assessment tasks/assignment, 
database of grades, records of attendance, observational notes, e-learning management 
system, other (specify). 

4. Data collection was snapshot (data collected one time), longitudinal (data captured from 
the same group of people at different points in time and relating to the same research 
question [s]), trend (data captured from different cohorts over a period of time using the 
same data collection instrument), pre-post (data captured before and after a particular 
intervention to measure the degree of change occurring as a result of that intervention), 
other (specify). 

5. Data were analysed using descriptive statistics (numbers summarising and describing 
observed data; e.g., through percentage, mean, median, standard deviation), inferential 
statistics (inferring properties about a larger population; e.g., through tests such as p-values, 
T-tests, ANOVAs), thematic analysis (coding of written text into themes), grounded theory 
analysis (explicitly mentioned by authors in the study), other (specify). 

6. Data was presented in tables, quotes, graphs, visual diagrams, case studies, themes, other 
(specify). 

7. Approach that best describes the article was qualitative, quantitative, mixed methods, other 
(specify). 

8. If research approach classified as mixed methods, then dominant approach was qualitative, 
quantitative, or equal use, other (specify). 

9. If research approach was classified as mixed methods, describe how qualitative and 
quantitative approaches are integrated across sections of the article: not integrated, partial 
integration (integrated in methods, results or discussion), or full integration (integrated 
across methods, results and discussion with conclusions draw from both sources of data). 
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RESULTS 

A total of 267 scholarly articles from three SoTL journals were reviewed. Of these, 89.55% 
(n=223) were suitable for inclusion in this study based on the parameters identified in the methods 
section. 


Demographics data 

Of the papers that we analysed, 82.51% (n=184) were published with first authors from USA 
followed by ‘other’ (8.97%, n=20), Canada (3.14%, n=7), Australia (2.69%, n=6), and the UK (2.69%, 
n=6). 

The vast majority of the articles, 89.24% (n=199), described research that was conducted at a 
single university; 6.28% (n=14) described cross institution SoTL within the same country; 3.59% (n=8) 
was cross-institutional from multiple countries; and 0.90% (n=2) was categorised as other (e.g. 
graduates, meta-analysis). 

Prevalence of qualitative, quantitative, and mixed-methods approaches 

Our analysis found 33.18% (n=74) of the papers analysed used a mixed methods approach, 
29.59% (n=66) used a qualitative approach, and 37.21% (n=83) used a quantitative approach. 

Gathering data, and ways of collecting and analyzing data 

The most common source of data were students (88.34%, n=197), followed by academics 
(21.08%, n=47). ‘Other’ data sources included professionals and employers, researchers outside of 
academia, graduates, the public, and teaching assistants (tutors in Australia and UK systems). The 
majority of studies used a single source of data (83.86%, n=187), with the rest using two or more 
sources. 

Data collection instruments varied across studies as shown in Table 2. The most common was 
surveys (71.30%; n=159). ‘Other’ included social network mapping, emails, website content such as 
comments and discussions, and teaching resources such as lesson plans. Of the 223 studies, 54.26% 
(n=12l) used multiple data collection instruments whilst the rest used just one. 

The majority of studies collected data only once as a snapshot (64.57%, n=144). A much lesser 
number of studies used a pre-post design (13.90%, n=3l), followed by longitudinal (8.52%, n=19) and 
trend (7.17%, n=16) approaches to collecting data (Table 2). 

Data analysis most commonly used was descriptive statistics (75.34%), followed by inferential 
statistics (54.71%) and thematic analysis (45.29%). The majority, 71.30%, of articles used multiple types 
of data analyses. Data presentation format was varied with data most frequently presented in tables or as 
quotes (Table 2). 

Qualitative and quantitative use, and integration in mixed-methods studies 

For studies classified as mixed methods research (n=74), we identified if one approach was 
predominantly used in these articles or whether there was a balanced use of both quantitative or 
qualitative approaches (following Hart et al., 2009). We found that there was fairly even distribution 
with a breakdown of predominantly qualitative (36.49%; n=27), predominantly quantitative (33.78%; 
n=25), and equal use of both quantitative and qualitative studies (29.73%; n=22). 

Where studies used a mixed methods approach (n=74), we analysed the papers to identify how 
the authors integrated both qualitative and quantitative research. The majority of papers used only 
partial integration of results as evidenced by 58.12% (n=43) of mixed methods studies presenting 
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qualitative and quantitative results and findings separately with an integrated discussion or conclusions 
section. To a lesser extent, 29.73% (n=22) used full integration where SoTL scholars integrated 
discussion, implication, and conclusions sections that clearly drew on both qualitative and quantitative 
findings. Just 12.16% (n=9) of mixed methods studies had no integration, whereby authors presented 
qualitative and quantitative results and findings separately with no integration between the two in the 
discussion or conclusions sections. 


Table 2. Summary statistics for all research articles (n=223) analysed by data collection instrument, data collection frequency, and 
data presentation style; split by research method. 



Total 

Mixed Methods 

Qualitative 

Quantitative 


N 

% 

N 

% 

N 

% 

N 

% 


Data Collection Instrument 


Survey 

159 

71% 

69 

43% 

27 

17% 

63 

40% 

Assessment task 

75 

34% 

24 

32% 

30 

40% 

21 

28% 

Grade database 

48 

22% 

11 

23% 

3 

6% 

34 

71% 

Interview 

45 

20% 

17 

38% 

28 

62% 

0 

0 

Focus groups 

31 

14% 

13 

42% 

16 

52% 

2 

6% 

Observational 

notes 

26 

12% 

9 

35% 

13 

50% 

4 

15% 

Reflection 

23 

10% 

9 

39% 

13 

57% 

1 

4% 

Other 

16 

7% 

6 

38% 

6 

38% 

4 

25% 

Attendance 

records 

4 

2% 

0 

0.00 

1 

25% 

3 

75% 

Online learning 
management 

2 

1% 

1 

0.50% 

0 

0 

1 

0.50% 


Data Collection Frequency 


Multiple Approach 

13 

6% 

6 

46% 

2 

15% 

5 

39% 

Trend 

16 

7% 

6 

38% 

3 

18% 

7 

44% 

Longitudinal 

19 

9% 

5 

26% 

9 

48% 

5 

26% 

Pre-Post 

31 

14% 

16 

52% 

1 

3% 

14 

45% 

Snapshot 

144 

64% 

41 

28% 

51 

35.41 

52 

36.11 


Data Presentation Style 


Table 

172 

77% 

62 

36% 

31 

18% 

79 

45% 

Quotes 

122 

54% 

62 

50% 

59 

48% 

1 

1% 

Graphs 

50 

22% 

14 

28% 

3 

6% 

33 

66% 

Visual/Diagram 

27 

12% 

10 

37% 

6 

22% 

11 

40% 

Case Study 

4 

1% 

1 

25% 

3 

75% 

0 

0% 

Themes 

111 

49% 

47 

42% 

60 

54% 

4 

3% 

Other 

34 

15% 

15 

44% 

3 

8% 

16 

47% 
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DISCUSSION 

In this study, we explored the use of quantitative, qualitative, and mixed-methods research in 
published SoTL literature over a three-year period in three explicitly focused SoTL journals. Our aim 
was to inform the debate about the research approaches used in SoTL by contributing data about the 
extent to which quantitative, qualitative, and mixed methods are implemented by SoTL researchers and 
scholars. 

Prevalence of qualitative, quantitative, and mixed-methods research in SoTL 

We found that quantitative approaches (37.21%) were the most prevalent in the published 
SoTL literature we examined. Mixed methods research drawing on both qualitative and quantitative 
approaches was identified in 33.18% of the articles in our analysis. With 29.59% of articles categorised as 
qualitative, the spread across the three research approaches was relatively evenly distributed. The 
distribution in the current study is similar to the Matthews, Divan, John-Thomas, Lopes, Ludwig, 
Martini, Motley and Tomljenovic-Berube (2013) study, which was expected, as both draw on published 
SoTL literature, although this study uses a more recent database of articles and uses a more clearly 
defined operational framework for classification of papers. Considered in light of the review of ten years 
of SoTL activity in one university, which found qualitative approaches (71%) dominated SoTL 
methodological approaches (Hubball et al., 2010), the relatively even distribution of results in the 
current study stands out. One explanation for the difference could be that the Hubball study represents 
an institutional culture whilst our study, spanning several SoTL journals, more accurately reflects the 
work of the SoTL community. 

Comparison between our SoTL results with reviews of methodological approaches from other 
fields of inquiry reveals stark contrasts as the prevalence of mixed method research varies considerably 
across disciplines. Hart, Smith, Swars and Smith (2009) reported that of the 710 research articles 
examined in mathematics education published over a period of 10 years, 50% used qualitative research 
design, 21% used quantitative, and 29% used a combination of qualitative and quantitative. Their 
findings clearly show that a single approach (qualitative) dominated publications in the field of 
mathematics education. Alise and Teddlie (2010) carried out a study to measure prevalence rates across 
a number of disciplines. These authors reported 24% of articles presented mixed methods research 
across five elite education journals, but only 6% in psychology and sociology, indicating further examples 
of uneven distribution of methodology across specific fields of inquiry. A striking study by van der Roest, 
Spaaij, and van Bottenburg (2015) demonstrated the scarcity of mixed methods research in the field of 
sports management, identifying only 1.7% of publications as using this approach. Whilst SoTL is having 
an internal debate about research study design with concerns about plurality of approaches, our results 
demonstrate a fairly even spread of approaches when compared to other fields of inquiry. The results of 
our current study support the ‘big tent’ claim that SoTL is inclusive of various approaches to research as 
espoused by Huber and Hutchings (2005). 

However, given the claims of several prominent SoTL scholars (Bass & Linkon, 2008), the 

description of SoTL research design should extend beyond those characterised as qualitative, 

quantitative, or mixed methods. For instance, alternative approaches are common in the humanities, 

such as close reading, which is a literary criticism technique involving the in-depth interpretation and 

analysis of written texts (Bass & Linkon, 2008; Chick et al., 2009a). However, our results from 223 

empirical SoTL articles did not find one instance of close reading, or other approaches that fell beyond 

the boundaries of qualitative or quantitative approaches. A reason for this could be that scholarship that 

employs discipline-specific research methods may be more readily found in disciplinary journals—in the 
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case of close reading methods, journals in the arts and humanities. By our operational definitions, any 
encountered articles utilising close reading would have been categorised as qualitative. Our findings 
therefore imply that SoTL has the scope to expand into new and alternative realms of research. New 
SoTL journals are explicitly inviting SoTL scholars to submit work that draws on diverse and under¬ 
utilised methodological approaches. For example, Teaching & Learning Inquiry: The ISSOTL Journal 
was established in 2013 and states in its Mission that, “its pages will showcase the breadth of the 
interdisciplinary field of SoTL in its explicit methodological pluralism, its call for traditional and new 
genres, and its international authorship from across career stages.” (Teaching & Learning Inquiry: The 
ISSOTL Journal, 2013, Mission section, para. l). As new SoTL journals emerge with a broader remit, it 
is plausible that future analysis of SoTL journals will yield results different than the current study. 

Gathering data, and ways of collecting and analyzing data 

Although approaches to SoTL research were found to be relatively evenly distributed across 
qualitative, quantitative, and mixed methods, in-depth examination of sources of data, data collection, 
and data analysis revealed disproportionate distribution. The vast majority of SoTL publications that we 
analysed gathered data from students (88.34%). Inquiry into student learning is the first of Felten’s 
(2013) five good practices for SoTL. Thus, the high percentage of data sourced from students is perhaps 
unsurprising as SoTL’s ultimate aim is to enhance student learning. However, the results from an 
analysis of abstracts submitted to the 2014 ISSOTL conference found less than 50% of the presentations 
or posters sourced data from learners (Powell et al., 2015). Of course, both studies involved gatekeepers 
in the form of peer-reviewers who decide what gets published, either in the journals selected for this 
study or the conference studied by Powell, et al. Differences in SoTL presented at a conference and 
published in international journals would expectedly vary. Nonetheless, further exploration of sources of 
data in SoTL would benefit the field. Given that we found 83.86% of SoTL publications in this study 
employed a single source of data, future research should examine the use of triangulation in SoTL 
studies given the consensus that triangulation improves research practices, particularly studies of 
learning and teaching that are messy and complex (Mathison, 1988). Triangulation could be through 
data gathered from multiple sources (learners but also other sources such as academics, employers or 
graduates) as appropriate. 

A range of data collection instruments was identified, although surveys were clearly a preferred 
tool that were used across qualitative, quantitative, and mixed methods studies. Powell, Harmon, and 
MacMillan (2015) highlighted the use of surveys in SoTL as an established means of collecting 
perception data. Although surveys were the dominant data collection mode found in the current study, a 
range of other instruments were identified (e.g. grades, reflective tasks, usage data from social media or 
online learning management systems) with just over 50% of the articles using at least two data collection 
instruments. This suggests that whilst SoTL scholars draw on a single source of data (mainly students), 
many are collecting relevant data in different ways. Our analysis also showed that studies typically drew 
on multiple ways of analysing (e.g. descriptive statistics, thematic analysis) and presenting data (e.g. 
tables, text, figures). Thus, some triangulation is evident. The range of data collection methods indicates 
variation that resonates with the pluralism of research approaches expected of such a broad field as SoTL 
(Huber & Hutchings, 2005). 

Analysis of data collection revealed another noteworthy trend. Of the 223 articles included in 

the study, 64.57% relied on one-off, or snapshot, collection of data. Matthews, Divan, John-Thomas, 

Lopes, Ludwig, Martini, Motley and Tomljenovic-Berube (2013) found that the 71% of 154 SoTL 

publications in their literature review were focused on single units of study (or modules). Given the 
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short time-frame of a single unit of study in many SoTL papers, the reliance on one-off data collection is 
perhaps unsurprising. However, capturing data over a period of time, either from the same participants 
or by comparing cohorts over multiple years, affords greater triangulation. There is clearly significant 
potential for SoTL studies to expand to gathering data over multiple points in time and in doing so 
strengthen the conclusions of their work. 

Qualitative and quantitative use and integration in mixed-methods studies 

Greene (2007) identifies three components as critical to the design of mixed methods research: 
the extent to which the qualitative and quantitative approaches are integrated, the relative prominence 
given to each approach, and the timing of data collection. Similarly, Leech and Onwuegbuzie (2009) 
describe mixed method research designs as comprising of three fundamental components: the level of 
mixing between the qualitative and quantitative parts (either partial or full), time orientation (qualitative 
and quantitative data collected concurrently or sequentially), and the emphasis afforded to the two 
approaches (equal status or dominant status). 

We found that approximately one-third of the papers categorised in our study as mixed methods 
used an equal-status design in which the same relative importance was given to the qualitative and 
quantitative approaches. In another third, the quantitative strand dominated, and in the final third, the 
qualitative strand dominated. Thus a fairly even distribution is evident within mixed methods studies 
classified in the SoTL articles analysed for this review. This supports the Huber and Hutchings (2005) 
‘big tent’ approach of inclusive SoTL approaches, at least within the types of mixed methods SoTL 
studies in this review. 

However, mixed methods researchers argue that the hallmark of a mixed methods study is equal 
integration between qualitative and quantitative approaches (Johnson, Onwuegbuzie & Turner, 2007). 
In practice, many researchers report difficulty in “bringing together the analysis and interpretation of the 
qualitative and quantitative data and [in] writing a narrative that links the analyses and interpretations 
together” (Bryman, 2007). An expanding number of publications are now advancing practical strategies 
or frameworks to facilitate better integration (Bazeley, 2012; Castro et ah, 2010; Heyvaert, Maes & 
Onghena, 2011; O’Cathain, Murphy & Nicholl, 2010). Ivankova, Creswell, and Stick (2006) propose 
using a visual representation of the research process, illustrating where, when, how, and why data sets are 
connected and mixed, to aid clarification of integration. Further exploration of mixed methods in SoTL 
would benefit the field, including SoTL-specific frameworks for employing mixed methods approaches. 

CONCLUSION 

By examining three-years of publications from three international, SoTL-focused journals, our 

paper offers insight into the types of SoTL research currently being used in the published literature and 

establishes evidence that will inform the ongoing debate about research design and shape future 

discussions in the field. However, our results should be considered in light of several limitations. First, 

choices used to define the body of literature under review inevitably introduced bias. By explicitly 

describing our rationale, article selection, and analysis procedures, we sought to provide readers with 

essential information about the objectives, scope, and criteria we used to facilitate our understanding 

and interpretation of the study’s results. Second, operationalising the varying approaches to complex 

research SoTL scholars employ, ones that are contested in the literature (e.g. qualitative, quantitative, 

mixed methods), into concrete definitions is inherently limiting. Again, we sought to address this by 

explicitly describing our definitions and analysis framework. Third, we deliberately avoided analysis of 

theoretical orientations found in our literature review and when discussing research approaches. We 
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acknowledge that all types of research maintain theoretical assumptions and orientations, regardless of 
whether or not they are explicitly stated. For pragmatic reasons (space in the article and time limitations 
of the authors) and research reasons (little known from which to build this study), we elected to focus 
on the current practical implementation of SoTL researchers. Future investigators would do well to 
further explore the theoretical elements of SoTL studies. 

Overall, this review of published SoTL literature reveals instances of methodological pluralism 
expected of the broad field that is SoTL, supporting the ‘big tent’ view of the field, particularly when 
compared to methodological reviews from other disciplines. The findings also demonstrate that SoTL 
can advance through the inclusion of methodological pluralism (e.g., use of close reading) and adoption 
of strategies known to enhance the quality of research (e.g., triangulation and visual representation). 
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